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Foreword 



The Center for Substance Abuse Treatment (CSAT) works to improve the lives of those 
affected by alcohol and other substance abuse, and, through treatment, to reduce the ill effects of 
substance abuse on individuals, families, communities, and society at large. Thus, one 
important mission of CSAT is to expand the availability of effective substance abuse treatment 
and recovery services. To aid in accomplishing that mission, CSAT has invested and continues 
to invest significant resources in the development and acquisition of high-quality data about 
substance abuse treatment services, clients, and outcomes. Sound scientific analysis of this data 
provides evidence upon which to base answers to questions about what kinds of treatment work 
best for what groups of clients, and about which treatment approaches are cost-effective methods 
for curbing addiction and addiction-related behaviors. 

In support of these efforts, the Program Evaluation Branch (PEB) of CSAT established 
the National Evaluation Data Services (NEDS) contract to provide a wide array of data 
management and scientific support services across various programmatic and evaluation 
activities. Essentially, NEDS is a pioneering effort for CSAT in that the Center previously had 
no mechanisms established to pull together databases for broad analytic purposes or to house 
databases produced under a wide array of activities. One of the specific objectives of the NEDS 
project is to provide CSAT with a flexible analytic capability to use existing data to address 
policy-relevant questions about substance abuse treatment. This report has been produced in 
pursuit of this objective. 

This report explores the potential uses of four large-scale follow-up studies: the 
California Drug and Alcohol Treatment Assessment (CALDATA); the Services Research 
Outcomes Study (SROS); the National Treatment Improvement Evaluation Survey (NTIES); and 
the Drug Abuse Treatment Outcome Study (DATOS). It discusses similarities and differences in 
study designs and methods, compares some program and patient characteristics, and 
systematically reviews the outcomes of treatment as revealed by each of the studies, concluding 
with implications and recommendations for further research with the four studies. 

Sharon Bishop 
Project Director 

National Evaluation Data Services 
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Abstract 



This report compares the research methods, provider and patient characteristics, and 
outcome results from four large-scale follow-up studies of drug treatment during the 1990s: the 
California Drug and Alcohol Treatment Assessment (CALDATA), Services Research Outcomes 
Study (SROS), National Treatment Improvement Evaluation Study (NTIES), and Drug Abuse 
Treatment Outcomes Study (DATOS). 

Methods and Characteristics. The first two studies were, respectively, state and 
national level stratified random probability samples of providers and patients using retrospective, 
one-contact designs. NTIES and DATOS were prospective, repeated-interview designs focusing 
on evaluations of urban provider networks with high percentages of minority group clients. Three 
of the studies covered approximately 1 -year before-treatment and after-treatment reference 
periods, and SROS covered 5-year reference periods. All studies achieved multistage response 
rates of about 60 percent, except DATOS, which achieved about 40 percent. 

Outcomes. From before treatment to after treatment, each study revealed across-the- 
board positive changes in drug use, crime, and health status. Clients in short-term and long-term 
residential treatment experienced positive changes in drug use, crime, employment, and health 
more often than clients in outpatient treatment. Clients discharged from (rather than maintained 
on) methadone displayed positive changes less often than any other group in treatment. Clients 
in SROS showed consistently fewer changes than the others on comparable variables, which may 
be due to its longer reference periods and lower precision of these comparable variables to 
measure long-term patterns of change. 

Implications for treatment research, policy, and practice. Researchers should explore 
further the potential of these large data sets to capture differences in client populations, 
organization of services, and clinical practices that may explain differential outcomes and costs. 
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I. Introduction 



Research advances on fundamental policy-relevant aspects of substance abuse treatment 
have arisen from a mixture of small experimental studies and large observational studies. Before 
the 1990s, large studies — which may be operationally defined as follow-ups of more than 1,000 
treated individuals under one protocol — were rare, occurring at roughly once-every- 1 0-years 
intervals (Simpson, 1974a, 1974b; Hubbard et al. 1989; Simpson and Curry, 1997). 

Nevertheless, these large studies shared enough in the way of common design features to 
generate important comparisons across time as well as answer new questions about treatment 
outcomes and their correlates. 

However, in the 1990s, not just one but four large-scale observational studies were 
performed: the California Drug and Alcohol Treatment Assessment, Services Research 
Outcomes Study, National Treatment Improvement Evaluation Study, and Drug Abuse 
Treatment Outcomes Study (best known by their acronyms: CALDATA, SROS, NTIES, and 
DATOS). These four studies were close enough in time and in design features to open the 
possibility of bringing the study results together in new ways. In this paper, we take some 
exploratory steps in this direction. In particular, we compare the major features and results of the 
four studies with each other for the first time, appraising thereby the feasibility of using these 
data to as an unprecedented observational laboratory, a 3 00-program/ 10,000-person source of 
substance abuse treatment follow-up data. 

This paper is based largely on published data, although some additional analysis was 
done on several unpublished variables to improve the comparisons. First, in a methodological 
comparison, we review the construction of the four study samples, including details about 
sampling, field procedures, and response rates. Next, we review characteristics of the programs 
selected and the demographic composition of the study samples. We compare the client outcome 
results of the four studies in terms of changes observed between intake and follow-up across a 
series of domains, including drug use, criminal activity, source of income, and physical and 
mental health. Finally, we develop some tentative implications of these methodological 
comparisons and outcome results for treatment research, policy, and practice. 
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II. Methodological Comparison 



From 1990 to 1997, four large-scale observational follow-up studies were begun and 
completed (through a first round of post-treatment follow-up interviews) to evaluate substance 
abuse treatment outcomes. The four studies were: 

■ CALDATA — California Drug and Alcohol Treatment Assessment (Gerstein et al., 
1994): a panel of 3,055 clients selected from client records abstracted in 87 clinical 
units, followed an average of 15 months after discharge from treatment in 1991-92; 
gross follow-up completion rate of 61 percent, multi-stage response rate of 58 
percent. 

■ SROS — Services Research Outcomes Study (Schildhaus et al., 1998): a panel of 
3,047 clients selected from client records abstracted in 99 units, followed up an 
average of 5 years after discharge from treatment in 1989-90; follow-up completion 
rate of 59 percent, multi-stage response rate of 55 percent. 

■ NTIES — National Treatment Improvement Evaluation Study (Gerstein et al., 1997): 
a panel of 6,593 clients completing intake interviews in 7 1 units, followed an average 
of nearly 1 year after discharge from treatment in 1993-94; follow-up completion rate 
of 82 percent, multi-stage response rate of 65 percent. 

■ DATOS — Drug Abuse Treatment Outcome Study, first-year follow-up (Simpson and 
Curry, 1997): a panel of 4,786 clients completing intake interviews in 76 units, 
followed up an average of 1 year after discharge from treatment in 1991-93; follow- 
up completion rate of 62 percent, multi-stage response rate response rate of 39 
percent. 

Exhibit II- 1 presents summary information on the study sponsors and performers, sampling 
methods, sizes, and follow-up rate calculations for each study. While the study designs are 
similar in many ways, such as how the follow-up questionnaires were designed and analyzed, the 
studies also differed in important respects. 

1. CALDATA 

The sample design of CALDATA, a study designed and performed for the State of 
California by the National Opinion Research Center (NORC) in collaboration with Lewin-VHI, 
was in some respects the most straightforward. The population sampled were clients who were 
discharged between October 1991 and September 1992 from treatment providers who received 
any public funding (through contracts with county substance abuse treatment agencies, the state 
Medicaid office [MediCal], or other public agencies) in California during that period. This 
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Exhibit II- 1 

Four Follow-Up Studies of Substance Abuse Treatment SDUs in the United States in the 1 990s 
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Exhibit II- 1 (Continued) 

Four Follow-Up Studies of Substance Abuse Treatment SDUs in the United States in the 1990s 
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population included a high percentage (more than 90%) of all licensed drug and alcohol 
treatment providers located in California. The approach was “cold follow-up,” that is, the clients 
were selected entirely from records obtained from treatment programs regarding specific 
treatment episodes. Based on information in the files (released for research purposes only, as 
permitted by state and Federal law), clients would then be sought, located, and recruited for 
interviewing about this past treatment episode and the periods before and after it. 

In the first stage of sampling, information contained in the California Alcohol and Drug 
Data System (CADDS) was used to select a probability sample of clinical service delivery units 
(SDUs), selected within strata of geographic region, county, and modality of treatment. The 
regions represented the different parts of California: Los Angeles County, with one-fourth of the 
state population; the other populous urban counties of Southern California; the San Francisco 
Bay area; the Central Valley; and the remaining sparsely populated mountain, desert, and coastal 
counties. Within these strata, 16 counties (of 58) were selected using systematic random 
sampling (the largest counties were selected with certainty, the others with probabilities 
proportional to size). From the aggregate of SDUs in these counties, 1 10 SDUs were picked, 
with probability proportionate to size, representing five types of treatment: methadone 
detoxification, methadone maintenance (separate licenses, although many units held both types), 
nonmethadone outpatient, residential units identified as employing the “California social model” 
of services; and other residential units, virtually all of which employed a long-term therapeutic 
community approach. 

Only 106 of these units, sampled from records, proved to have actually treated patients in 
the 1-year reference period, and these 106 were part of 97 treatment provider organizations. The 
provider organizations with more than one selected SDU turned out mostly to offer both 
methadone detoxification and methadone maintenance in the same clinic with the same staff. 
Moreover, about half of these dual-SDU units were linked together with other sampled units 
owned by a handful of proprietary “chains.” 

In the second stage of sampling, 87 SDUs (82 cooperating providers) permitted 
CALDATA staff to randomly select eligible clients for follow-up from their clinical records. 
CADDS made it possible to estimate the numbers of eligible clients in noncooperating as well as 
in cooperating sample providers, while data collected during the record abstraction phase made it 
possible to later compare respondents with nonrespondents within cooperating sample providers, 
a comparison that indicated great similarity between the interviewed (responding) and 
noninterviewed groups. 
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CALDATA staff randomly selected and abstracted 3,055 records from the 87 cooperating 
SDUs, with sampling procedures to generate approximately equal numbers (about 30 cases) from 
each unit, except that in units with fewer admissions than this, all admissions were selected, and 
in the very largest units, double or in some cases triple samples were drawn to reduce weighting 
variability. All clients who were discharged from treatment between October 1991 and 
September 1992, including those who were admitted but received no treatment services, were 
eligible for participation in the study. The sample was further supplemented to include clients 
who were in methadone maintenance during the eligibility period and were still in the same 
episode of treatment at the time that records were abstracted in early 1993. The 9-month 
interviewing field period began in April 1993 and ended in December 1993. 

At the conclusion of the field period, 1,858 sample clients had been interviewed; 
however, due to project deadlines, only 1 ,826 cases could be included in the published analyses 
and data files. 1 The postdischarge follow-up durations at the time of interview ranged from 9 to 
24 months with a median of 1 5 months. CALDATA therefore completed interviews with 61 
percent of all sampled cases (64% excluding the deceased), and the multi-stage response rate is 
estimated to be 58 percent (see Exhibit II- 1 footnote). 

2. SROS 

Like CALDATA, SROS was a cold follow-up study that used probability sampling at 
each stage of sample selection. Just as CALDATA was the first full probability sample of 
substance abuse treatment clients followed up at a state level, SROS was the first national level 
follow-up study to employ probability sampling of providers and clients. However, the SROS 
sample of cooperating providers, comprising 99 treatment facilities that were in operation 
between September 1989 and August 1990, did not represent the general population of treatment 
providers and clients in the U.S. as comprehensively as the CALDATA sample represented 
California. The main reason is that SROS, carried out by NORC for the Substance Abuse and 
Mental Health Services Administration and fielded during a 9-month span of client interviewing 
in 1995-1996, was based on a sample of treatment facilities that had participated in the Drug 
Services Research Study (DSRS) in 1991, a study performed for the National Institute on Drug 
Abuse by Westat, Inc., in collaboration with Brandeis University. The sampling rules that had 



The four studies did not vary in one key respect: all provided the same monetary incentive of $15 for completing 
a follow-up interview. Three of the four studies also collected urine samples at the time of follow-up ($10 
incentive), but at different sampling rates: SROS in three-fourths of all cases, NTIES in one-half, DATOS in 
one-quarter of cases, and CALDATA in no cases. 
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been used to select DSRS facilities from the 1990 NDATUS census of providers excluded 
outright just over half (50.4%) of the listed providers in NDATUS, namely all those classified as 
providing treatment for “alcohol only” rather than “drug only” or “combined drug and alcohol,” 
and all those providers who were missing data on this (or on other) key design variables. 
Moreover, of the 146 facilities originally selected for DSRS, 26 facilities did not participate in 
DSRS and thus were ineligible for SROS, and another 21 providers who participated in DSRS 
did not participate in SROS, in many cases having gone out of business in the interim and 
leaving no custodial deposition of treatment records. 

Like the CALDATA provider sample, the SROS sample was stratified by region of the 
country and included outpatient methadone providers (both detoxification and maintenance), 
outpatient nonmethadone providers, longer term residential treatment providers, (most of them 
therapeutic communities), and shorter term residential treatment providers. Unlike CALDATA, 
these shorter term providers across the U.S. were largely hospital inpatient programs, that is, 
“medical model” in staffing and orientation rather than “social model.” The DSRS sample had 
been selected without regard to size of SDU, so the numbers of clients selected in the larger 
SDUs were adjusted to reflect SDU size. 

Clients in SROS, as in CALDATA, were followed on a cold basis, that is, strictly from 
records-generated information, 5 to 6 years after leaving treatment, compared with an average of 
1 5 months after treatment in CALDATA. The SROS identifying and locating information was 
again restricted to the information contained in abstracted clinical treatment records, and the 
SROS client follow-up rate of 59 percent, translating to 65 percent of living sample cases, is 
remarkably similar to that of CALDATA, as is the estimated multi-stage response rate of 55 
percent. However, the cost per case in SROS was substantially higher than CALDATA, 
requiring about half again as many field interviewer hours per completed interview. 

Aside from the difference in resources expended, the similar results probably reflect the 
less urbanized character of the SROS sample and the tendency of more poorly organized 
programs (those with the least informative records) to be omitted from the initial sampling frame 
or to become lost to the sample during the intervening years. In addition, the much longer lead 
time of the SROS project, a result of slower stage-by-stage approval processes, permitted various 
locating efforts such as electronic search for database matches to proceed in advance of rather 
than relatively late in the respective 9-month field periods. Finally, since SROS client data 
collection was performed by the same survey organization as performed CALDATA, the 
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experience previously gained with this cold follow-up methodology probably benefitted the 
second study. 

3. NTIES 

The sample of substance abuse treatment programs included in NTIES was drawn on a 
purposive rather than probability basis from a special cohort of SDUs. The 800 or so eligible 
SDUs were affiliated with one or more of 157 successful 1990-91 applicants to the Center for 
Substance Abuse Treatment (CSAT) for demonstration grants to enhance or expand treatment 
services for selected population groups, including individuals residing in nine of the largest 
urban centers (“target cities”), public housing residents, racial/ethnic minorities, pregnant and 
postpartum women, and adolescent and adult criminal justice populations. CSAT contracted 
with NORC and its subcontracting partner, Research Triangle Institute (RTI), to design and carry 
out NTIES. The NTIES staff selected a set of SDUs that would assure coverage of each of the 
major demonstration program areas, provide economical geographic clustering of the field cases, 
and select SDU sizes in favor of larger ones to control research costs and minimize potentials for 
SDU attrition across the multi-year field period due to financial distress. 

Unlike CALDATA and SROS, research subjects in NTIES were recruited to the study at 
the time of intake to treatment, so that follow-up was based on collecting research-oriented 
locator information as well on program records. Within the 71 productive sample SDUs in 16 
states, all clients who met two requirements were eligible for follow-up: 1) any receipt of 
treatment services, operationalized as either staying a minimum of one night if a residential 
program or completing one outpatient treatment visit beyond the intake procedure, if an 
outpatient program; and 2) completing a 75-minute research intake interview that included the 
detailed locating information to be used for follow-up within 21 days after being admitted to 
treatment between August 1993 and October 1994. Except in the largest SDUs, where the roster 
of clients was randomly subsampled to cap the maximum number per SDU at about 200 cases, 
all clients beginning treatment in each sample SDU during the 1 5-month “admission window” 
were recruited for an NTIES intake interview, on average about 1 1 0 per SDU. 

Of those eligible for NTIES, 85 percent completed the intake interview, with most of the 
loss reflecting failure to successfully schedule the intake interview within 21 days of admission 
rather than to an outright refusal. All of the 6,593 clients who completed the intake interview 
were targeted for follow-up interviews approximately 12 months after leaving treatment. In the 
interim, all 6,593 clients were also eligible for a ‘treatment experience” interview at the time of 
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discharge or after being in treatment for an extended period, and 80 percent of the NTIES panel 
completed the interim interview. 2 

The 12-month follow-up response rate was 82 percent (slightly higher, but rounding to 
the same number, when the small number of deceased and other excluded cases are removed 
from the denominator), about 20 points higher than the follow-up interview completion rates 
obtained in CALDATA and SROS. NTIES required about one-third less follow-up interviewer 
time per completed follow-up interview than CALDATA and nearly two-thirds less than SROS. 
However, the difference in interviewer time between CALDATA and NTIES disappeared when 
one took into account the time required to complete the NTIES intake as well as follow-up 
interviews. NTIES staff pursued in-person follow-up if the respondent resided within a 150-mile 
travel radius. Also, like CALDATA, NTIES permitted telephone interviewing where personal 
interviews could not be obtained, and telephone interviews accounted for 2 percent of NTIES 
follow-up cases. 

The response rate advantage of NTIES over SROS and CALDATA was due largely to the 
prospective enrollment of the sample at the time of admission, so that the follow-up rate is based 
only on cases for whom research-quality locator data have been collected, and on cases that have 
already complied to some extent with the research protocol. The higher follow-up rate may also 
have been partially due to the characteristics of programs included in the specialized target 
population; in particular, correctional programs achieved response rates exceeding 90 percent. 
However, the targeting of CSAT grants toward “needier” programs having less secure 
relationships with their funding agencies and clients, and the concentration of sample SDUs in 
inner city areas, which are a recurrent challenge for fieldwork, would not favor higher follow-up 
rates. Moreover, if one bases the NTIES follow-up rate not on those completing the intake 
interview but on all those eligible for the intake interview, the follow-up rate becomes 70 percent 
of the eligible nondeceased sample, which is much closer to the SROS and CALDATA results; 
and the estimated multi-stage response rate is 65 percent. 3 



2 The principal reason for noninterview was, again, missing the window of eligibility, which was within 8 weeks 
of discharge. In outpatient programs in particular, information about discharge was often not obtained or 
confirmed soon enough to locate and recruit the client before this window expired. 

3 The NTIES field period for follow-up interviews was approximately 12 months; however, cases were released to 
follow-up at different points, with some made available at 10 months after treatment with eligibility nominally 
ending at 14 months; others as early as 5 months after treatment due to the need to conclude the study. The 
median interview took place 1 1 months after discharge, and more than 90 percent were completed between 6 
and 15 months after treatment. 



^ J:\62 1 050\NORC\PROSPECT\PROSPECT.WPD 




19 



NEDS, July 19, 1999, Page 9 



Methodological Comparison 



A final element in the NTIES follow-up experience was an evident “house effect.” The 
NTIES SDUs were divided among two sets of field assignments. One set of research SDUs was 
staffed and supervised by one of the two NTIES survey houses (RTI), the second set was staffed 
by the other house (NORC). The follow-up interview completion rate in the first set of SDUs 
was 70 percent, as compared to 85 percent in the second set. Regional patterns of field 
assignment would not account for this difference, since the NORC assignment included Eastern 
inner cities; both assignments included some Southern and Southwestern cities; and the RTI 
assignments covered mostly smaller urban areas and towns in the West. 

NTIES drew from short-term residential/hospital, long-term residential (with treatment 
plans greater than 2 months and nearly all calling themselves therapeutic communities), and 
methadone and nonmethadone outpatient SDUs. Relative to the other studies, NTIES had fewer 
methadone maintenance programs, drawing only about 8 percent of its client panel from such 
programs, as compared with more than 20 percent in each of the other studies (see Exhibit II-2); 
however, the other studies included far more programs in order to fill out their methadone strata 
than the quantitative share this modality of treatment actually occupies in the national picture 
(about 12-13%; see Gerstein et al., 1997 and SAMHSA, 1997). NTIES was also the only study 
that included correctional facilities, drawing about one-fourth of its client sample from such 
facilities. 

4. DATOS 

Like NTIES, DATOS featured a purposive sample of drug and alcohol treatment 
programs in which the follow-up research cohort was recruited on a prospective basis. In 
DATOS, 1 1 cities were initially chosen as sites for the study, many of them the same cities (and 
some of the same programs) as had been studied a decade earlier in the large-scale TOPS cohort 
(Hubbard et al., 1989). DATOS, like the Treatment Outcome Prospective Study (TOPS), was 
sponsored by the National Institute on Drug Abuse and performed by RTI. 4 Within each 
DATOS city, an attempt was made to recruit “typical and stable” programs from each of four 
modalities, including short-term and long-term residential (similar to the SROS and NTIES 
definitions), outpatient methadone, and outpatient drug-free (nonmethadone). 



4 However, whereas the TOPS data collection was carried out by RTI with NORC as a subcontractor, the DATOS 
fieldwork was performed entirely by RTI. Beginning in 1996, the analytic phase and a 5-year DATOS follow-up 
wave were taken over by a different set of research organizations under the Cooperative DATOS (C-DATOS) 
agreement; see Simpson and Curry, 1997). 
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Relative to the other three studies, an important distinguishing characteristic of DATOS 
is that the eligibility criteria for follow-up of clients within cooperating programs were stringent 
and complex. The follow-up sample was limited to clients who completed two 90-minute intake 
interviews and were from one of the 76 programs in which 20 or more clients had completed 
both of two 90-minute DATOS intake interviews. In addition, the subsample selected for 
follow-up (4,786 clients) was selected so as to oversample longer lengths of stay in treatment. 

As in NTIES, some characteristics of the DATOS sample may have facilitated the 
locating of sample cases and therefore favored a higher follow-up response rate. These 
characteristics included the restriction of the follow-up to organizationally stable providers; 
intermediate research interviews for those remaining in treatment, scheduled at 1, 3, 6, and 12 
months in treatment. DATOS also undersampled clients with shorter lengths of stay, whose 
compliance can be more difficult to obtain. 5 As with NTIES, not all individuals admitted to 
treatment in the participating SDUs entered the research sample. Specific information is not 
available for DATOS (as it was for NTIES) on what percentage of the eligible patients completed 
the first intake interviews. Interviews were conducted only in-person and within the 1 1 cities, 
even when sample clients had moved to other cities or nonmetropolitan areas. 

The purposive nature of the first-stage sample and the selective noncooperation and 
ineligibility of sample programs reduce the generalizability or external validity of DATOS 
information. Simpson and Curry (1997) report that 120 cooperating treatment programs were 
originally selected within the 1 1 cities. (The number of programs selected but refusing to 
cooperate is not reported.) Twenty-four of these 120 programs were dropped from DATOS early 
on due to low initial client flow, while 20 more were excluded from the follow-up protocol 
because they yielded fewer than 20 clients who completed both intake interviews. 

The DATOS 12-month follow-up response rate was 62 percent, quite similar to the 
response rates for the cold follow-up in SROS and CALDATA percent, but about 20 percentage 
points lower than the comparable NTIES statistic. This rate falls to 48 percent of the total 
nondeceased/nonincapacitated client denominator, compared to 70 percent in NTIES, and due to 
the dropping of so many smaller SDUs, the estimated multi-stage response rate falls to 39 
percent. The DATOS protocol was less aggressive than NTIES in seeking follow-up interviews; 



5 Using the criterion of remaining in treatment for 3 months or longer (very much appropriate to three of the four 
DATOS treatment types, less so for the short-term inpatient modality), this difference is visible in response rates 
among those selected for follow-up. In the long-term residential modality, 62 percent of respondents versus 50 
percent of nonrespondents surpassed this length of stay; in the outpatient drug-free mode, 58 versus 48 percent; 
in methadone, 92 percent versus 78 percent. 
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nevertheless, DATOS required approximately 25 percent more interviewer hours per completed 
follow-up interview than NTIES. 

5. CONCLUSION 

There were large variations in follow-up completion rate and multi-stage response rate 
among the four treatment studies. These variations were due partly to variations in basic 
research design and partly to a difference in aggressiveness or effectiveness in carrying out 
follow-up fieldwork. The single interview “cold” follow-up design, operating on a purely 
retrospective basis, is attractive because it can be completed much more rapidly than the 
prospective/retrospective, two-interview design. The trade-off for more rapid study completion 
is probably some loss in precision due to recall factors, a reduction in total information due to the 
reduced total interview time, and a loss of about 20 percentage points in response rate relative to 
a pre-enrolled panel; but only about 5 points loss over the total client sample at intake. 
Considering the time required for the pre-enrollment interview, and considering that the on-site 
effort required to generate a records-only sample is approximately equal to the cost of 
completing a post-discharge client record abstraction (required in the NTIES and DATOS 
protocols), there is not a substantial difference between the two design types in terms of field 
hours (and associated cost) for comparable sample sizes for similar post-discharge periods. 

Two of the studies were based on probability samples of well-defined and geographically 
comprehensive general treatment populations. Even though probability sampling of general 
treatment populations poses challenges for gaining cooperation from an adequate proportion of 
sampled SDUs and for successful follow-up of former treatment clients, this kind of sampling 
has many attractive features. Probability sampling supports inferences to a population that 
endures in time and remains politically as well as scientifically meaningful — e.g., the population 
of individuals admitted to drug and alcohol treatment in a specified geographic domain during 
specified periods. Probability samples in successive studies provide the firmest foundation for 
generalized conclusions about changes in treatment effectiveness, since they minimize the risk 
present in purposive samples that differences in the research results are artifacts of the different 
SDUs that happened to be studied. 6 



6 DATOS did return to largely the same cities — in many instances the same programs — that had been studied 10 
years earlier in TOPS (Hubbard et al., 1989), allowing some valuable temporal comparisons of treatment 
components and effectiveness. 
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In the absence of a common target population, it is somewhat hazardous to compare 
response rates among studies. Since the response patterns of each survey may reflect the unique 
population that was represented, it is not surprising that some generalizations about differences 
between respondents and nonrespondents are not supported by more than one survey. For 
example, CALDATA and SROS reported higher response rates among Hispanics than 
nonHispanics, while DATOS reported the opposite and NTIES, with its higher overall responses 
rates, found no differences. CALDATA, SROS, and DATOS report consistently higher follow- 
up rates among women than among men, whereas in NTIES (again, with higher overall response) 
the difference by gender was quite small. In view of the differences in sampling methodology 
and coverage, it may be expected that there would be differences in provider and patient 
characteristics and in outcomes achieved. The next sections of the paper are devoted to 
exploring these expectations. 



J:\62I050\NORC\PROSPECTAPROSPECT.WPD 



NEDS, July 19, 1999, Page 14 



III. Published Data on Provider and Patient Characteristics 



Each of the four treatment studies had a provider survey component, allowing in principle 
for comparisons of various types of treatment modalities. These sections were completed by 
directors and other personnel of the selected treatment facilities. The surveys contain items that 
reflect similar elements — sources of revenue, treatment goals and emphases, ownership 
charactaeristics, as well as staffing patterns, caseloads, and services provided. However, the 
published results of the four studies emphasized different aspects of providers; so much so that 
only one published measure was comparable across as many as three of the studies. We have 
calculated some additional measures here to permit some additional comparisons across three of 
the studies, but this discussion only reflects the beginning stages of working through the provider 
data in search of comprehensive comparisons. 

1. SOURCES OF REVENUE 

Reports on SROS and DATOS included measures of the providers’ sources of revenue, 
specifically, the percentage of total revenue that was public revenue. To be classified as public 
revenue, it was necessary for part of reported finances to have originated from Federal, state, or 
local government agencies rather than from private insurance or other sources. Exhibit II-2 
presents the published findings on this measure for SROS and DATOS. The percentage of total 
revenue from public sources was fairly consistent for each treatment type. About three-quarters 
of the revenues of SDUs were public. This appears to be lower in the methadone and outpatient 
SDUs in the nationally representative SROS data than in DATOS. 

2. CLIENT INVOLVEMENT IN TREATMENT GOALS 

Three of the four studies reported on items in their questionnaires that represented 
whether treatment participants helped “to a great extent” to establish their own treatment goals. 
These data are presented in Exhibit II-2 and may be summarized as follows: 

■ Providers of short-term residential treatment in NTIES (33%) reported a much lower 
percentage of clients participating in setting their own treatment goals than such 
providers reported in CALDATA (77%) and SROS (67%) 

■ NTIES methadone providers reported less extensive client goal-setting (53%) than 
providers in CALDATA (82%) and SROS (74%) reported 
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■ The percentage of clients in long-term residential facilities reported by administrators 
to have helped set their own treatment goals appeared to be higher in SROS (78%) 
than in CALDATA (53%) or NTIES (51%) 

■ For outpatient non-methadone facilities, the percentage of clients reported to set their 
own treatment goals is more comparable than among the three studies: CALDATA at 
43 percent, SROS at 54 percent, and NTIES at 56 percent. 

3. STABILITY OF OWNERSHIP OR ADMINISTRATION 



Measures of stability included in CALDATA, SROS, and NTIES asked whether the 
treatment unit had undergone any change in ownership or administrative oversight for a defined 
period of time: CALDATA covers nearly 2 years, SROS 5 years, and NTIES 1 year. Exhibit 
II-2 lists the results. 

■ About half (45%) of short-term residential providers and one-third (33%) of 
outpatient methadone providers in SROS reported change in ownership or 
administrative oversight between 1990-1994. The comparable percentages were 22 
percent and 23 percent for CALDATA and 18 percent and 21 percent for NTIES, 
consonant with the shorter exposure periods in these studies. 

■ Despite the differing exposure periods, the percentage of turnover reported by long- 
term residential providers was quite consistent across the three studies, ranging from 
26 percent to 29 percent. 

■ The outpatient nonmethadone results also seem contrary to the exposure periods: the 
turnover was lowest in NTIES (13%), somewhat higher in SROS (19%), and highest 
in the CALDATA units (29%). 

4. DEMOGRAPHICS OF CLIENTS 



Exhibit III-l presents an overview of the demographic composition of the client samples 
of each of the four studies. Most of the modality-by-modality comparisons yield rates of gender 
and age distribution within about 15 percentage points of one another. However, a few 
differences exceeded this, and one characteristic in particular, racial/ethnic distribution, was 
systematically different across the samples. NTIES and DATOS, drawing heavily from medium 
to large metropolitan areas, had consistently higher percentages of respondents who are black or 
Hispanic than do SROS and CALDATA. Generally, the “minority” groups were about two- 
thirds of participants in NTIES and DATOS and about 40 percent in CALDATA and SROS. 
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Exhibit III-l 

Demographic Variables by Study and Treatment Modality 
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Published Data on Provider and Patient Characteristics 



In addition, across modalities, NTIES consistently had lower percentages of respondents 
with a high school degree than either SROS and lower percentages of respondents who were 
married. NTIES focused on programs that had received CSAT grants and were either in the 
nation’s largest cities, working with “critical populations” including people in public housing, or 
working with criminal justice populations. 

5. CONCLUSIONS 

The differences in the ownership/administrative turnover, client goal-setting, and client 
demographics suggest that the provider and client populations selected in the four studies may be 
distinctive in important domains. The very limited degree of overlap in the published provider 
data precludes extensive interpretation of the differences, but the results do suggest the 
importance of carrying out a more comprehensive assessment that would go well beyond the 
published materials. The long-term residential providers in NTIES and CALDATA and the 
outpatient nonmethadone units in CALDATA seemed to turn over more rapidly than expected 
relative to the SROS sample. The meaning of these differences awaits further analysis of the 
provider information. 
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IV. Drug Use Outcomes 



Over the years that drug treatment outcome studies have been conducted, a variety of 
approaches have been taken to describe drug use. Until the 1970s, virtually all of the treatment 
programs under study focused on heroin addiction, and little attention was paid to other drugs. 

In the first major national survey, the Drug Abuse Reporting Program (Sells, 1974a, 1974b), 
elaborate efforts were made to classify drug users according to the combinations of drugs used. 
These classification efforts continued but grew less complicated (incorporating fewer, broader 
types), in the Treatment Outcome Prospective Study (Hubbard et al., 1989). In the studies of the 
1990s, drug use had been treated two ways. First, the most commonly used drugs — heroin, 
cocaine, marijuana, alcohol, amphetamine — have been viewed separately, each one indexing a 
conceptually distinct change. Secondly, individuals have been asked to identify their “primary” 
drug or drug-combination, and analysis has focused on changes in use of the primary drug. In 
this paper, we focus on the individual drugs that the published results have treated in similar 
ways, with threshold measures of consumption during the reference periods before and after 
treatment. These drugs are marijuana, cocaine (powder and crack), and heroin. 

Before reviewing the relevant data, it is useful to review once again the treatments being 
compared and the way they are handled in the analysis. The short-term residential units are, at 
first glance, the most diverse category. All short-term residential units in CALDATA are 
“California Social Model” programs; those in SROS and DATOS are “hospital inpatient” units; 
and NTIES enrolled a mixture of hospital-based wards, spiritually guided Salvation Army type 
programs, and social -model oriented programs. The long-term residential units in all studies are 
largely therapeutic communities. The methadone units are similar to one another in that all are 
FDA-licensed providers; however, the patients selected for follow-up and how they are grouped 
in the analyses differ. In DATOS, all methadone patients were merged together — those who 
were still in treatment at the time of follow-up and those who had been discharged. In SROS, all 
of the patients were discharged but some from detoxification and others from maintenance 
treatment plans. In NTIES, all patients had been enrolled in maintenance and none in 
detoxification plans; however, those who stayed continuously in treatment through the follow-up 
interview are analyzed separately from those who did not. Finally, in CALDATA, clients 
continuing in maintenance through the follow-up period are analyzed separately from those who 
were discharged, but the latter four include both those admitted to detoxification and those who 
were discharged from maintenance plans. 

Within each study, the same population was included in both the before-treatment and 
after-treatment statistics. The population sizes analyzed and reported in the following pages vary 
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Drug JJse Outcomes 



somewhat from those reported in Exhibit II- 1. The DATOS and SROS populations are equal to 
the follow-up cohorts in Exhibit II- 1, that is, 2,966 and 1,799 cases, respectively. The 
CALDATA population is slightly smaller than the follow-up cohort (1,826), because 32 of 1,858 
cases were collected too late to be coded for inclusion in the analysis. The NTIES group 
reported on here comprises 3,702 out of 5,388 in the follow-up cohort. It is so much smaller 
mainly because we omit here the entire correctional treatment modality (there being no 
equivalent modality in the other three studies). We also omitted a small group of persons whose 
follow-up interviews were completed before or after the 5- to 16-month “window” of eligibility 
for follow-up. 

1. MARIJUANA 

The CALDATA, SROS, and NTIES reports were based on using marijuana five times or 
more during the reference period, while DATOS was based on using marijuana at least weekly. 

As shown in Exhibit IV- 1, in all modalities in all studies, marijuana use declined 
significantly (using a t-test to determine significant differences) in the reference period after 
treatment. In each study, the change measures in the two residential treatment modalities were 
quite similar. They were also very similar to each other across CALDATA, NTIES, and DATOS 
(-51% to -65%), and notably lower in SROS (-35% and -32%). Change in the outpatient 
methadone and nonmethadone units was more disparate, particularly in DATOS where the 
methadone clients reported relatively low levels of change (-19%) and the nonmethadone clients 
reported high levels of change (-67%). In NTIES the outpatient modalities were very similar 
(-42%), and in SROS and CALDATA they were not far apart (ranging from -19% to -34%). 

We also developed statistics for marijuana in CALDATA using a measure that is similar 
to the reported measure in DATOS. The CALDATA measure is how often marijuana was used 
during the month when it was used most often, we specifically report here whether it was used 
more than 5 days per month; this is compared with DATOS 1 time a week or more during the 
past year. The results, as shown in Exhibit IV-2, are much more like DATOS in terms of the 
absolute levels of use before and after treatment and are somewhat closer as well in terms of 
relative change. 
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Exhibit IV-1 

Treatment Effects on Marijuana Use by Study and Treatment Modality 
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Drug Use Outcomes 



Exhibit IV- 2 

Treatment Effects on Marijuana Use in CALDATA and 

DATOS 





CALDATA 


DATOS 


Used more than 5 days in the 
month of most frequent use 
during the reference period. 


Which number on this card 
indicates how often you used 
[DRUG] during the past year? 
[Weekly/1 time a week OR 
MORE] 


Time Frame: 


1 year 


1 year 


Treatment Effect 


PRE 


POST 


% A 


PRE 


POST 


% A 


Short-Term 

Residential 


29.2 


10.5 


-64 


30.3 


10.5 


-65 


Long-Term 

Residential 


39.1 


14.4 


-63 


28.3 


12.7 


-55 


Methadone 

Discharge 


21.5 


15.1 


-30 


17.1 


13.9 


-19 


Methadone 

Maintenance 


18.1 


15.8 


-13 


Outpatient 

Non-Methadone 


32.7 


17.0 


-48 


25.4 


8.5 


-67 



Notes: Differences are significant at p<.05 level, unless stated otherwise. 



2. COCAINE 

DATOS combined powdered cocaine and crack use in its published analyses, while the 
other studies separated results for these forms of cocaine. Exhibit IV-3 reports the combined 
cocaine results for DATOS along with the powdered cocaine results for the other three studies. 
As with marijuana use, all modalities in all studies report significant declines in cocaine use. As 
with marijuana use, the residential modalities yielded results similar to each other in each study 
and quite similar across the four studies (-47% to -69%), with SROS reporting changes at the 
lower end of the range (-47% and -55%). The studies also reported very similar results in the 
outpatient modality (-51% to -56%), with SROS somewhat lower (-42%). There was a wider 
range in the methadone results, with NTIES, DATOS, and the CALDATA maintenance group 
similar (-45%, -48%, and -36%), SROS and CALDATA discharge group lower (both -20%). 
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Exhibit IV-3 

Treatment Effects on Cocaine Use by Study and Treatment Modality 
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Drug Use Outcomes 



Exhibit IV-4 reports for three studies the results for crack cocaine. All residential and 
outpatient results for NTIES and CALDATA were quite similar (-42% to -52%), but the SROS 
results were markedly lower (no significant change on outpatient, -14% and -32% in residential). 
There were also no significant changes in crack use in methadone users in NTIES or SROS, but 
these were significant in CALDATA (-28% and -42%). Crack use was at a much lower level of 
use before treatment in the all modalities than was cocaine or marijuana use. 

3. HEROIN 

As shown in Exhibit IV-5, pretreatment heroin use ranged from rather uncommon to 
moderately common (prevalence of 6% to 27%) in all modalities except methadone, where it was 
extremely common (82% to 98%). Changes in heroin use were significant in all modalities in 
three of the four studies; in SROS, only in the methadone modality was there significant change. 
Outside of methadone programs, the outpatient changes were lower than in residential treatment, 
and the CALDATA changes were lower (-26% to -36%) than those in NTIES and DATOS (- 
41% to -69%). Discharged methadone patients changed less (-13% to -27%) than those in 
continuing maintenance (-39% to -51%), and the most change was reported in DATOS (-69%). 
As with marijuana, we calculated for CALDATA a measure similar to the DATOS measure (was 
used at least 5 times/month in the “peak” month, compared with 1 time/ week or more during the 
past year). As reported in Exhibit IV-6, these comparisons bring the outpatient and maintenance 
results in CALDATA and DATOS somewhat closer together. 
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Drug Use Outcomes 



Exhibit IV-4 

Treatment Effects on Crack Use by Study and Treatment Modality 





CALDATA 


SROS 


NTIES 




During [months], did you use 
crack 5 times or more? 


Looking at the calendar, in the 
5 years before [date] did you 
use crack 5 times or more? 


Since (DATE) have you used 
crack, not cocaine powder, 
five times or more? 


Time Frame: 


1 year 


5 years 


1 year 


Treatment Effect 


PRE 


POST 


% A 


PRE 


POST 


% A 


PRE 


POST 


% A 


Total 


29.1 


15.8 


-467 


29.0 


24.3 


-17 


50.4 


24.8 


-51 


Short-Term 

Residential 


41.4 


18.5 


-55 


36.5 


31.2 


-14 


60.3 


30.0 


-50 


Long-Term 

Residential 


49.5 


24.7 


-50 


48.5 


33.8 


-32 


65.9 


32.6 


-51 


Methadone 

Discharge 


20.2 


14.6 


-28 


24.2 


23.7 


n.s. 


32.4 


27.1 


n.s. 


Methadone 

Maintenance 


9.7 


5.6 


-42 








24.3 


29.6 


n.s. 


Outpatient, 

Non-Methadone 


20.0 


11.7 


-42 


14.0 


13.7 


n.s. 


45.6 


22.0 


-52 



Notes: Differences are significant at p<.05 level, unless stated otherwise. 
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Drug Use Outcomes 



Exhibit IV-6 

Treatment Effects on Heroin Use in 
CALDATA AND DATOS 





CALDATA 


DATOS 


Used more than 5 days in the 
month of most frequent use 
during the reference period. 


Which number on this card 
indicates how often you used 
[DRUG] during the past year? 
[Weekly/1 time a week OR 
MORE] 


Time Frame: 


1 year 


1 year 


Treatment Effect 


PRE 


POST 


% A 


PRE 


POST 


% A 


Short-Term 

Residential 


8.1 


4.6 


-43 


7.0 


2.2 


-69 


Long-Term 

Residential 


12.1 


8.0 


-34 


17.2 


5.8 


-66 


Methadone 

Discharge 


89.1 


68.8 


-23 


89.4 


27.8 


-69 


Methadone 

Maintenance 


75.3 


29.6 


-61 


Outpatient 

Non-Methadone 


7.8 


4.5 


-42 


5.9 


3.3 


-44 



Notes: Differences are significant at p<.05 level, unless stated otherwise. 
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V. Other Outcomes 



1. CRIMINAL ACTIVITY 

The pre-treatment and follow-up instrumentation for all four studies contain measures 
of criminal behavior. These questions assess the following general types of criminal behavior: 



■ Criminal activity that may or may not have resulted in criminal justice 
involvement 

■ Criminal justice involvement, including arrests, convictions, incarcerations, etc. 

■ Income from illegal activities. 

The published findings for each of the four studies assess changes in criminal behavior as a 
treatment outcome. In each case, there were significant reductions in various criminal behaviors 
following discharge from treatment as compared to pre-treatment behaviors. 

■ CALDATA Significant decline in the percentage of clients arrested and 
committing any illegal activity, as well as specific illegal activities, including 
drug sale/manufacturing, burglary, and assault 

■ SROS Significant reduction in the commission of several criminal activities, 
including vehicle theft, DUI/DWI, drug sale/manufacture, prostitution and 
procurement, fraud/forgery, theft/larceny, burglary, and assault 

■ NTIES Significant decline in the percentage of clients arrested, for both any 
charge and for drug possession; the percentage of clients committing specific 
criminal activities, including drug sale/manufacture, shoplifting, and assault; and 
the percentage of clients supported primarily with money obtained through illegal 
activities 

■ DATOS Significant reduction in arrests, incarcerations, criminal justice 
involvement, and the commission of predatory illegal activity (i.e., assault, 
robbery, burglary, larceny, forgery, or dealing in stolen property). 

Exhibit V-l summarizes the published findings from CALDATA, SROS, NTIES, and DATOS 
on the percentage of clients arrested, booked, or taken into custody during the periods before and 
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Other Outcomes 



after treatment. The percentage of clients reporting arrests during the pre- and post-treatment 
reference periods in CALDATA, NTIES, and DATOS, overall and within each treatment 
modality, are similar. By contrast, the percentage of clients reporting arrests during these periods 
in the SROS study is substantially higher; most likely also as a result of the longer reference 
period. In CALDATA and NTIES, the reductions in arrests in the modalities outside of 
methadone were all in the range of -60 percent to -68 percent. By contrast, the declines in 
DATOS were appreciably smaller (-31% and -40%), and the declines in SROS were smaller yet 
(-9% to -28%). Among methadone clients, the maintenance groups in CALDATA and NTIES 
reported the largest reductions in arrests (-82% and -55%), with lesser reductions in the 
CALDATA and NTIES discharge groups and the least change in the DATOS methadone groups. 
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Exhibit V-l 

Treatment Effects on Arrests by Study and Modality 
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Exhibit V-2 summarizes the published findings from CALDATA, SROS, and NTIES 
on the percentages of clients who sold illegal drugs before and after treatment. (Note: DATOS 
included a comparable measure; however, the data for this measure have not been published.) In 
each study, there was a reduction in the percentage of clients who sold drugs during the reference 
period, both overall and within treatment modalities. Although pretreatment drug selling was 
two to three times more common in the NTIES study, the NTIES and CALDATA changes were 
very similar in the residential units ( -75% to -80%) and in the methadone groups, albeit the 
discharged clients changed less (-55 and -65%) and maintenance clients changed more (-86% 
and -96%). The CALDATA outpatient clients had the least drug selling before treatment and 
changed less than the NTIES outpatients (-57% versus -79%). The SROS clients on the whole 
reported about half the level of reductions compared with CALDATA and NTIES. 

2. EMPLOYMENT AND WELFARE 

All studies collected data on the extent and duration of full-time employment, although 
the published DATOS measure was full-time employment throughout the reference period while 
the other three studies measured any full-time employment, yielding commensurately higher 
absolute figures (Exhibit V-3). There was more variability in these results among studies and 
treatment modalities than in any other outcome dimension measured, with CALDATA’s results 
the worst and DATOS’s the best; and methadone discharge the worst and long-term residential 
the best. More specifically, the CALDATA and SROS residential groups reported no significant 
employment changes, while in NTIES and DATOS, there were moderately positive changes in 
the short-term units (10% and 16%) and more appreciable ones in the long-term units (32% and 
85%). In the outpatient groups, CALDATA reported negative change (-1 1%), SROS no change, 
and NTIES and DATOS positive change (18% and 34%). In a similar vein, the SROS and 
CALDATA discharge groups reported negative changes (-34% and -25%), the CALDATA 
maintenance group no change, the NTIES methadone groups small positive change (5%), and the 
DATOS group a larger positive change (34%). 



EKLC 

ifliflaffHHaiEB 



J:\62 1 050\NORCVPROSPECT\PROSPECT.WPD 



47 



NEDS, July 19, 1999, Page 31 



Other Outcomes 



Exhibit V-2 

Treatment Effects on Drug-Selling by Study and Modality 
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Exhibit V-3 

Treatment Effects on Full-Time Employment by Study and Modality 
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best copy available 



Other Outcomes 



CALDATA and NTIES collected and reported before/after data on receipt of welfare 
income (Exhibit V-4). There was little change in CALDATA; only the decline in methadone 
discharge clients (-13%) was significant. All of the NTIES groups changed significantly, but by 
modest amounts, ranging from a +6 percent increase in short-term residential to a -19 percent 
decrease in outpatient units. 

3. PHYSICAL AND MENTAL HEALTH 

All studies collected extensive data on physical and mental health, but only a limited 
base is available in the published results. CALDATA and SROS reported on changes in general 
health status using a standard four-point scale (from poor to excellent); NTIES and DATOS 
reported on those with limitations in work or other activities (see Exhibit V-5). The largest 
changes were reported in CALDATA, the smallest in SROS and NTIES, with DATOS in 
between. In the CALDATA residential units, despite the “social” rather than medical emphasis 
of the short-term modality, only half as many clients reported fair or poor health after treatment 
as before. The other studies reported declines of -12 percent to -1 8 percent in most cases, but no 
change in SROS long-term residential and a change of -3 1 percent in the DATOS short-term 
residential. In methadone, CALDATA and NTIES discharge groups reported no change, with 
other groups reporting -13 to -21 percent change in those with fair/poor health or disabling health 
conditions. Outpatient results ranged from no change in NTIES to -41 percent change in 
disabling conditions in DATOS. 

Exhibit V-6 displays results from three studies concerning receipt of inpatient 
psychiatric treatment during the reference periods. The results are quite varied. There was only 
one statistically significant change in NTIES, a reduction in inpatient hospitalization among the 
outpatient drug treatment group. There were more changes in CALDATA, where the two 
residential groups reported reductions of -30 percent and -44 percent while the methadone 
discharge group reported a change of -53 percent. In SROS, two groups changed but in the 
opposite direction: the long-term residential and outpatient groups reported increased overnight 
mental health care (39% and 42%). 

Rather than the mixed picture of results on mental health treatment, there was 
uniformity in the important mental health status measure of suicidal thoughts or attempts, 
reported in all four studies (Exhibit V-7). Nearly all groups reported significant reductions, 
excepting only the NTIES methadone groups where there was no change. The most substantial 
changes were among the NTIES short-term residential and both CALDATA residential client 
groups (-58%, -63%, -64%), with lower levels of change in the DATOS methadone group (- 
24%), and all others between -33 percent and -54 percent. 
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Exhibit V-4 

Treatment Effects on Welfare Income by Study and Modality 
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Exhibit V-5 

Treatment Effects on General Health Status by Study and Modality 
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Exhibit V-6 

Treatment Effects on Psychiatric Hospitalization by Study and Modality 
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Exhibit V-7 

Treatment Effects on Suicide Attempts by Study and Modality 
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VI. Implications for Treatment Research, 
Policy, and Practice 



In this report we have taken an exploratory look at three aspects of the four large-scale 
studies of drug treatment outcomes that have taken place in the 1990s, namely the following: 

■ Methodological features including sample design, measurement, follow-up 
procedures, and response rates 

■ Treatment program and client population characteristics 

■ Outcome results, as measured by before/after differences on key dimensions of 
substance use and related behavior. 

Because this is an exploratory study, it would be inappropriate to draw any direct conclusions for 
policy and practice from these results. However, the immediate research implications and 
downstream potential to inform policy and practice are very important. 

There are several main methodological features of note. Two of the studies employed 
stratified random samples of treatment units and clients, interviewing clients once on a 
retrospective basis, while the other two are purposive, prospective, and involve repeated 
measures over time. The most visible result of these differences in terms of population 
parameters is that the purposive samples, reflecting a long-term Federal emphasis in its funding 
patterns and outcome research, have significantly more minority group members than the 
representative samples: roughly two-thirds rather than 40 percent. If there were substantial 
ethnic- or race-related differences in response to treatment, this would comprise a significant 
issue in generalizing from the results of purposive studies. Although study results that are not 
the direct focus of this report indicate significant differences by race and ethnicity in some 
pretreatment characteristics, particularly primary drug, there do not appear to be substantial 
enough differences in the response to treatment by corresponding population subgroups to assign 
high concern to this matter. Nevertheless, future studies must remain alert to departures from 
representativeness, an issue discussed in more detail in a companion NEDS paper (Johnson, & 
Gerstein, 1999) 

Three studies have fairly similar reference periods of about a year before and after 
treatment for most outcome measures, while the fourth, SROS, covers much longer periods of 5 
years before and after — although it does include numerous measures that index the past year 
only, similar to the other studies. The particular variables best suited to measure before/after 
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Implications for Treatment Research Policy , and Practice 



change in SROS are evidently different from those appropriate to the shorter term studies, which 
makes direct comparisons of such changes more difficult. However, SROS, with its lengthier 
time span and national probability sampling frame, should prove valuable as a calibrator or 
adjuster for some of the results of the other studies. SROS can provide, for example, a useful 
way to evaluate the core analytic assumption that, in the absence of treatment, adult respondents 
would display fairly stable population parameters on key outcome variables between the 1-year 
reference periods — in effect, the null hypothesis with regard to behavioral change in the absence 
of treatment. 

A 5-year follow-up data collection has recently been completed with a subset of the 
DATOS respondents, and comparisons between the 5-year DATOS and SROS data would be 
very useful. In the same vein, longer term follow-ups of CALDATA, which has the most 
complete sampling frame, or NTIES, which has the best follow-up and multi-stage response 
rates, would substantially enrich the possibilities of examining treatment careers and treatment 
outcomes with both short-term and long-term measures of change. 

Comparably computed measures of response rates indicate remarkably similar multi- 
stage sample completion rates for three of the studies, with NTIES a notch above the other two, 
but an appreciably lower rate for the fourth study, DATOS. All of the field work for the studies 
was performed separately or in combination with two research organizations, RTI and NORC. 
Both organizations have published experience in performing large-scale, geographically 
distributed substance abuse treatment outcome studies. DATOS was the only one of the four 
carried out entirely by the first research organization, but a similar difference between the two 
organizations was recorded in the jointly performed NTIES. The explanation for the lower 
DATOS rates lies partly in design decisions but appears mostly to reflect less aggressive or 
effective follow-up procedures rather than differences in resource levels or any other 
circumstances of the research. An important issue for further comparison is whether these 
differential completion and response rates lead to any differences in results beyond the inevitable 
reductions in precision that result from smaller follow-up sample sizes. 

Across the outcome domains of drug use, crime, income, and health, several findings 
strongly invite further intensive study. Clients in short-term and long-term residential treatment 
appear to experience positive changes in drug use, crime, employment, and health more often 
than do clients in outpatient treatment modalities. Since populations of residential clients differ 
in many respects from outpatient groups, this result does not necessarily translate to intrinsic 
advantages in program quality or effectiveness in residential (and inpatient) treatment settings. 
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Implications for Treatment Research, Policy, and Practice 



Nevertheless, this result addresses a very important debate about the allocation of resources to 
different types of treatment, and demands more extensive study across these data sets, which are 
especially rich in both nonmethadone outpatient and long-term residential programs. 

Clients who have been discharged from (rather than maintained on) methadone appear, 
in general, to display positive changes less often than any other group in treatment. This confirms 
widely accepted information about this well-studied type of treatment. More detailed combing of 
data on methadone SDUs and clients in these four studies may shed new light on the reasons why 
such a large percentage of methadone patients are discharged rather than maintained. 

Finally, employment and welfare utilization appear to change less often than any other 
outcome measures. These outcome measures are no doubt the least central of all the ones 
reviewed here to the mission and service focus of most treatment programs, and they are perhaps 
the most affected by systematic external variations such as the business cycle and public policy 
decisions. Nevertheless, further attention is needed to understand these results, such as division 
of the treatment population by pretreatment work experience and related economic variables 

In general, the findings of this exploratory analysis suggest that at least three of the 
studies are comparable enough to permit continued joint exploration, including the possibility of 
pooling cases to some extent to permit more sophisticated multivariate studies. As the public use 
files of all the sets become available, researchers will not be limited to comparing published 
items but can use the full range of outcome and process variables available in the respective sets 
of research instruments. The addition of a DATOS 5-year follow-up to SROS, and the possibility 
of adding longer term follow-up data on the other study cohorts provide additional opportunities 
to turn these research investments into knowledge that can inform policy and practice. 
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Appendix 



Description of the National Treatment Improvement Evaluation Study and 
Center for Substance Abuse Treatment Demonstrations (1990-1992) 

The National Treatment Improvement Evaluation Study (NTIES) was a national 
evaluation of the effectiveness of substance abuse treatment services delivered in comprehensive 
treatment demonstration programs supported by the Center for Substance Abuse Treatment 
(CSAT). The NTIES project (1992-1997) was designed and performed for CSAT by the National 
Opinion Research Center at the University of Chicago with assistance from Research Triangle 
Institute. The NTIES project collected longitudinal data between FY 1992 and FY 1995 on a 
purposive sample of clients in treatment programs receiving demonstration grant funding from 
CSAT. Client-level data were obtained at treatment intake, at treatment exit, and 1 2 months after 
treatment exit. Service delivery unit (SDU) administrative and clinician (SDU staff) data were 
obtained at two time points, 1 year apart. 

1. THE NTIES DESIGN 

1.1 The Administrative/Services Component 

The NTIES study design had two levels — an administrative or services component and a 
clinical treatment outcomes component. The administrative component was designed to assess 
how CSAT demonstration funds were used, what improvements in services were implemented at 
the program level, and what kind and how many programs and clients were affected by the 
demonstration awards. Four data collection instruments were used to gather administrative/ 
services data: the NTIES Baseline Administration Report (NBAR), the NTIES Continuing 
Administrative Report (NCAR), the NTIES Exit Log, and the NTIES Clinician Form (NCF). 

The unit of analysis for the administrative component was the SDU, defined by CSAT as 
a single site offering a single level of care. The classification of level of care is based on three 
parameters: 

■ Facility type (e.g., hospital, etc.) 

■ Intensity of care (e.g., 24-hour, etc.) 

■ Type of service (e.g., outpatient, etc.). 
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An SDU could be a stand-alone treatment provider, or it could be one component of a multi- 
tiered treatment organization. For example, a large, county mental health agency may be the 
organization within which the SDU is located. The organization may have multiple substance 
abuse treatment components, such as a county hospital and a county (ambulatory) mental health 
center. The county hospital may have multiple SDUs, such as an inpatient detoxification service, 
an outpatient counseling service, and a hospital satellite center providing transitional care. In 
summary, the SDU provided NTIES evaluators with a stable, uniform level of comparison for 
examining service delivery issues. 

A range of key clinician-specific data elements (within the administrative component) 
were assessed using the NCF. The NCF items were an important adjunct to the facility- (SDU) 
level instruments; these items assessed clinician training, experience, client exposure, and service 
provision, and were completed by all counseling and clinical (medical and therapeutic) staff at 
the individual SDUs. 

1.2 Clinical Treatment Outcomes Component 

The unit of analysis for the clinical treatment outcomes component was individual client 
data. NTIES measured the clinical outcomes of treatment primarily through a “before/after” or 
“pre- to post-treatment” design. This method compares behaviors or other individual 
characteristics in the same participants, measured in similar ways, before and after an 
intervention. 

Information about clients’ lives for the before period were obtained from the NTIES 
Research Intake Questionnaire (NRIQ), which was administered sometime during the clients’ 
first 3 weeks of treatment. The specific areas assessed included: 

■ Drug and alcohol use 

■ Employment 

■ Criminal justice involvement and criminal behaviors 

■ Living arrangements 

■ Mental and physical health. 
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Information about clients’ lives for the after period were obtained from the NTIES Post- 
discharge Assessment Questionnaire (NPAQ), with the same areas assessed at roughly 12 
months post-treatment. Other client data sources included a treatment discharge interview 
(NTIES Treatment Experience Questionnaire, NTEQ), abstracted client records, urine drug 
screens collected at the time of the follow-up interview, and arrest reports from state databases. 

1.3 The Outcome Analysis Sample 

Between August 1993 and October 1994, research staff successfully enrolled 6,593 
clients at 71 SDUs to participate in three waves of an in-person, computer-assisted data 
collection protocol. These SDUs were chosen from the universe of treatment units receiving 
demonstration grant funding from CSAT. Some of the selected facilities were wholly supported 
by CSAT awards, while others received only indirect support or none. 

Clients were interviewed three times: shortly after admission on their first day of 
treatment, when they left treatment, and 12 months after the end of treatment. Less than 10 
percent of the eligible clients refused or avoided participation, and more than 83 percent of the 
recruited individuals (5,388 clients) completed a follow-up interview. Additional sample 
exclusions included: 

■ Missing or undetermined treatment exit date 

■ Inappropriate length of follow-up interval (less than 5 or more than 16 months) 

■ Clients incarcerated for most or all of the follow-up period (nearly all had been 
treated while incarcerated, and were not yet released). 

The additional sample exclusions resulted in a final outcome analysis sample of 4,41 1 
individuals. 

2. TREATMENT DEMONSTRATION PROGRAMS 



CSAT initiated three major demonstration programs and made 157 multi-year treatment 
enhancement awards across 47 states and several territories during 1990 through 1992. One 
objective common to all demonstrations was CSAT’s emphasis on the provision of 
“comprehensive treatment” services to targeted client populations. The recipients of these 
awards focused special attention on the substance abuse treatment service needs of minority and 
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special populations located primarily within large metropolitan areas. The demonstration 
programs are briefly described below. 
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2.1 Target Cities 

Under this demonstration, nine metropolitan areas were selected to receive awards, of 
which half were included in the NTIES purposive sample. The following treatment improvement 
activities were explicitly provided for in the awards: 

■ Establishment of a Central Intake Unit (CIU) with automated client tracking and 
referral systems in place 

■ Provision of comprehensive services, including vocational, educational, biological, 
psychological, informational, and lifestyle components 

■ Improved inter-agency coordination (e.g., mental health, criminal justice, and human 
service agencies) 

■ Services for special populations — adolescents, pregnant and postpartum women, 
racial and ethnic minorities, and public housing residents. 

2.2 Critical Populations 

Under this demonstration program, awardees were required to implement “model 
enhancements” to existing treatment services for one or more of the following critical 
populations: racial and ethnic minorities, residents of public housing, and/or adolescents. 

Special emphasis was given to services provided to the homeless, the dually diagnosed, or 
persons living in rural areas. A total of 1 30 grants were awarded, covering services such as 
vocational support/counseling, housing assistance, integrated mental health and/or medical 
services, coordinated social services, culturally directed services, and others. 

2.3 Incarcerated and Non-Incarcerated Criminal Justice Populations 



Under this demonstration program, funds were directed toward improving the standard of 
comprehensive treatment services for criminally involved clients in correctional and other 
settings. Some program emphasis was placed on ethnic and/or racial minorities. Nine 
correctional setting demonstrations were funded: five in prisons, three in local jails, and one 
across a network of juvenile detention facilities. All projects included a screening component to 
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identify substance-abusing inmates, a variety of targeted treatment interventions (e.g., therapeutic 
communities, intensive day treatment programs), and a substantial aftercare component. 

A total of 10 non-incarcerated projects were funded. Five programs targeted 
interventions at clients in diversionary programs, three focused services on probationers or 
parolees, and two programs targeted both populations. Almost all of the funded demonstration 
projects included the following components: 

■ Basic eligibility determination, followed by systematic screening and assessment 

■ Referral to treatment 

■ Graduated sanctions and incentives while in treatment 

■ Intensive supervision in treatment 

■ Community-based aftercare with supervision and service coordination. 

In total, 19 criminal justice projects were funded as part of the CSAT 1990-1992 demonstrations, 
and as indicated in the next section, these projects were purposively over-sampled in order to 
obtain a more robust evaluation of this program. 

3. DESCRIPTION OF SDUS AND CLIENTS BY TREATMENT MODALITY AND 

PROGRAM TYPE 

The 71 SDUs contributing clients to the outcome analysis sample are characterized by 
modality and (demonstration) program type in Exhibit A-l below. Among the 698 SDUs in the 
NTIES universe: 52 percent (n=365) were Target Cities programs, 39 percent (n=274) were 
Critical Populations programs, and 9 percent (n=59) were Criminal Justice programs. 

In terms of the SDUs sampled for the NTIES outcome analysis, 44 percent were Target 
Cities programs, 38 percent were Critical Populations programs, and 23 percent were Criminal 
Justice programs. Criminal Justice SDUs were purposely over-sampled as part of the NTIES 
evaluation design (CSAT, 1997). Nearly half of the sampled SDUs were (non-methadone) 
outpatient programs, and about one-quarter were long-term residential programs. 
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Exhibit A-l 

SDUs in the Outcome Analysis Sample 


Program Title 

Number of SDUs 
(% of NTIES 
Universe) 1 


NTIES 

Sample 


Methadone 


Outpatient 


Long-Term 

Residential 


Short-Term 

Residential 


Correctional 


Target Cities 

n=365 (52%) 


31 

(44%) 


6 


15 


6 


4 


0 


Critical 

Populations 

n=274 (39%) 


27 

(38%) 


1 


13 


10 


3 


0 


Criminal Justice 

n=59 (9%) 


13 

(23%) 


0 


5 


0 


0 


8 


Totals 

N=698 (100%) 


71 

(100%) 


7 


33 


16 


7 


8 



Exhibit A-2 

Distribution of Clients in the Outcomes Analysis Sample 


Program Title 












Number of Clients 






Long-Term 


Short-Term 




(% of Analysis Sample) 


Methadone 


Outpatient 


Residential 


Residential 


Correctional 


Target Cities 


377 


1,214 


504 


505 


0 


n=2,600 (59%) 


(89%) 


(78%) 


(60%) 


(58%) 




Critical Populations 


45 


220 


298 


368 


0 


n=931 (21%) 


(11%) 


(14%) 


(35%) 


(42%) 




Criminal Justice 


0 


132 


39 


0 


709 


n=880 (20%) 




(8%) 


(5%) 




(100%) 


Totals 












n=4,41 1 (100%) 


422 


1,566 


841 


873 


709 




1 The original NTIES universe of SDUs included a program type called Specialized Services. Because clients for 
the outcome analysis sample were not drawn from these SDUs (n=94), they are excluded from the Exhibit. 
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As shown in Exhibit A-2, 59 percent of all NTIES clients were sampled from Target Cities 
SDUs. Slightly over 21 percent of all NTIES clients were sampled from Critical Populations 
SDUs, and 20 percent were sampled from Criminal Justice SDUs. Outpatient (non-methadone) 
SDUs treated over one-third (35%) of the clients in the outcomes analysis sample, and almost 80 
percent of these were sampled from Target Cities programs. 

Readers who are interested in more detailed information about the NTIES project are 
invited to visit the NEDS Web site at: http://neds.calib.com. The NEDS Web site provides the 
full-length version of the NTIES Final Report (1997), as well as copies of all data collection 
instruments employed in NTIES. 
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